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Abstract Body 



Background/context: 

Proficiency in algebra correlates with future academic and economic achievement (Atanda, 

1999; Berkner & Chavez, 1997), yet a great many students find success in algebra elusive (Ball, 
2003; Helfand, 2006; Kollars, 2008; Meehan & Huntsman, 2004; Rubin, 2007; Silver, Saunders, 
& Zarate, 2008). Although it seems that developing this proficiency sooner rather than later is 
important, for many students this does not seem to be an option, in part, because they lack the 
understanding of the basic arithmetic concepts on which algebra will build (National 
Mathematics Advisory Panel, 2008). 

At least one of the reasons cited for this lack of understanding and the difficulty students often 
encounter when transitioning from arithmetic to algebra is that many teachers of mathematics 
seldom build on students’ prior pre-conceptions, understanding, intuition, or innate problem 
solving strategies (Donovan & Bransford, 2005). Instead they often present math as a set of 
rules, procedures and facts (Bransford, Brown, & Cocking, 1999); present mathematics in a 
seemingly random or disorganized manner; and divorce procedural knowledge from what the 
processes or results mean (Fuson, Kalchman, & Bransford, 2005). In fact, our experience and a 
good deal of research suggest that, in the courses that precede algebra, instruction in the United 
States primarily focuses on the learning and application of algorithms or “shortcuts” that students 
can use to solve a ( particular ) problem (Ma, 1999; Novak, Mintzes, & Wandersee, 2000; Saxe, 
Gearhart, & Seltzer, 1999; Sternberg et al., 2000). 

Increasingly, research in three areas - expert versus novice learning, learning research in 
cognitive science, and research into the use of formative assessment - suggests how we might 
improve the way mathematics is taught. 

Work as far back as deGroot (1965), and continuing through Chi and her colleagues (Chi, 

Bassok, Lewis, Reimann, & Glaser, 1989; Chi & Slotta, 1993) into the present (Fuson et al., 
2005; Kaput, 2004; VanLehn, Siler, Murray, Yamauchi, & Baggett, 2003), hints at the 
importance of helping teachers present a more “expert-like” conceptualization of domains like 
arithmetic. Expert-novice research emphasizes that a common distinction between experts and 
novices is that experts have well connected webs of knowledge formulated around key 
foundational ideas (e.g. natural selection in biology and plate tectonics in geology). Novices, on 
the other hand, organize domains around superficial aspects of a problem and approach problem 
solving by trying to recall a memorized algorithm based on those superficialities (Chi & Roscoe, 
2002). It is apparent from our experiences that many elementary and middle school math 
teachers (and their students) organize their understanding as novices. Coupled with the literature, 
this suggests to us that organizing the primary and middle school mathematics domain around 
key foundational algebraic concepts, especially concepts already understood by students, would 
improve both teacher pedagogy and student success in algebra and in mathematics as a whole. 

Cognitive science research also suggests the importance of conceptual understanding before 
learning algorithms or procedures if we want students to build on existing knowledge in order to 
abstract key principles (Kester, Kirschner, & vanMerrienboer, 2005; Rittle- Johnson & Alibali, 
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1999; Siegler, 2003; Vendlinski, 2009). Research suggests that many students have little 
understanding of what procedures mean or why they work (Fuson et al., 2005; Kilpatrick, 
Swafford, & Findell, 2001) and, consequently, struggle when required to abstract concrete 
representations to broader applications or situations. The literature and our research suggests 
that, in the United States, instruction prior to algebra focuses largely on the teacher transmission 
and student application of problem solving routines (American Educational Research 
Association, 2006; Zevenbergen, 2005). Often neither student nor teacher is aware of how or 
why the algorithm works, what the limitations of the algorithm might be, and how the algorithm 
is connected to concepts or knowledge the student has previously mastered (examples include 
“combine like terms” , “invert and multiply”, and “whatever you do to one side, you do to the 
other”) (Fuson et al., 2005; Ma, 1999; Vendlinski et al., In press). Here again, connecting the 
development and meaning of such procedures with prior knowledge seems key to improving 
both teacher and student understanding. 

Finally, knowing what to teach the students in front of them requires that teachers accurately 
assess current student knowledge and understanding, provide appropriate feedback to the student 
and create or modify instruction to help move the student to the desired goal. The research 
literature is clear that the use of formative assessment can significantly improve student 
performance (Black, Harrison, Lee, Marshall, & Wiliam, 2003; Black & Wiliam, 2004; Black & 
William, 1998; Wiliam & Thompson, 2007). Yet our research just as clearly suggests that 
teachers have difficulty providing appropriate feedback to students and, depending on the topic, 
many teachers don’t know how to use formative assessment results to modify instruction to 
promote student learning or proficiency in mathematics (Heritage, Kim, Vendlinski, & Herman, 
In press). Clearly, improving teacher understanding of student learning is key to improving 
student mathematical proficiency. 

Based on this literature and experience, we see our professional development challenge as 
improving the ability of teachers to “help students build and consolidate prerequisite 
competencies, understand new concepts in depth, and organize both concepts and competencies 
in a network of knowledge.” (Donovan & Bransford, 2005, p.232). While the ultimate measure 
of change in teacher quality for these efforts is improved student proficiency in algebra, the fact 
that such improvements will occur two or three years in the future and is mediated by a number 
of intervening teachers suggests a more proximal measure of teacher quality is required to 
determine the immediate effects of our professional development efforts. 

Our conceptualization of what makes an effective teacher has important implications for how we 
improve and assess teacher quality. Among other things, a model based on the research 
described above suggests that merely evaluating a teacher’s ability to recall procedures is likely 
to lead to inadequate or inaccurate (invalid) inferences about teacher quality. In fact, Nathan, 
Koedinger, and Martha (2001) and others (e.g. Shulman, 1986) have concluded that both content 
knowledge and pedagogical content knowledge must be assessed in order to make quality 
inferences about teachers. Still, such assessments of teacher knowledge have proven difficult 
(Tittle, 2006), although not impossible to develop (Hill & Ball, 2004). As Tittle (2006) points 
out, “assessment methods [in studies of teacher learning and development] are evolving from 
work within the expert / novice paradigm . . . and using analyses of content. . .” (p. 956). With this 
in mind, reserachers at the National Center for Research on Evaluation, Standards, and Student 
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Testing (CRESST) have built on the work of Leinhardt and Ball and their colleagues (see Ball, 
Lubienshi, & Mewbom, 2001; Leinhardt & Greeno, 1984; Leinhardt & Smith, 1985) to 
developed a means to measure teacher ability to conceptualize foundational ideas and to use 
information from formative assessment to plan or modify instruction (Heritage et al., In press). 

Purpose/objective/research question/focus of study: 

Our hypothesis is that if teachers (as experts) understand and teach concepts from the position of 
expertise teacher quality will improve. We believe that focusing on the key ideas will deepen 
both teacher and student understanding and allow learners to build the concepts necessary to 
form solid foundations for the application of mathematics both in and out of school. But such 
methods of instruction often require that teachers learn to reorganize and teach fundamental 
concepts in more expert-like ways. Because many teachers were taught in a system that stressed 
procedure rather than conceptual understanding, and the dearth of guidance on how teachers 
might change their teaching (Kieran, 2003) in this manner, such change may be difficult. 

Building on the expert-novice literature (Chi et al., 1989), and the findings by Carpenter, 
Lenamya, Levi, Lranke, and Empson (2000) using cognitively guided instruction, and research 
on formative assessment (Black et al., 2003; Herman et al., 2008; Marzano, 2006; Wiliam, 

2006), we developed a program of professional development designed to help teachers 
conceptualize mathematics instruction and improve their ability to effectively use formative 
assessment data in the 6 th grade math classroom. Both the formative assessments and the 
professional development were designed around a collection of big ideas developed by content 
experts and math educators (Niemi, Vallone, & Vendlinski, 2006). The program of professional 
development supports a larger formative assessment effort as part of a US Department of 
Education sponsored program called PowerSource. 

In particular, the professional development builds on three of the approximately twenty 
foundational concepts experts suggest are the most important aspects of the algebra domain. 
These three concepts are: 1) the multiplicative identity and its application to rational 
equivalency; 2) the understanding of multiplication as specifying a number of identical groups as 
applied in the case of distribution; and 3) the understanding of the equal sign as an equivalence 
operator in solving equations. 

We combined the knowledge map of Leinhardt as refined by Chung and colleagues (Chung et 
al., 2006; Leinhardt & Smith, 1985) and a validated inferences from a student work task 
developed by Heritage and Vendlinski (Heritage et al., In press; Heritage & Vendlinski, 2006) to 
measure teacher quality (content organization and pedagogical ability). We used these as pre- 
post measures to determine changes in the teachers between the start and end of professional 
development in order to test our hypotheses and to measure the effects of professional 
development. These combined measures were designed to assess how each teacher organized a 
limited part of the domain of algebra and that teacher’s ability to use student feedback in 
planning instruction. 

Our ultimate goal is to demonstrate that the professional development methods we advocate 
improve student proficiency in algebra. Initially, however, we must demonstrate that our efforts 
actually improve near-term teacher content knowledge organization and have demonstrable 
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effects on teacher ability to use formative assessment results. The results from these efforts are 
reported here. As next steps, we will observe actual teacher pedagogical practice and evaluate the 
proficiency of students in the classrooms of teachers who have actively participated in the 
PowerSource program. 

Setting: 

The 9.25 hours of professional development was conducted largely outside of school hours at the 
district office or at one of the school sites within each district. The initial four hours of PD was 
almost always done prior to the beginning of the academic year. In the two cases where this was 
not possible, substitutes were provided for teachers during the school day. All 90 minute follow- 
up sessions with the teachers were conducted in after school settings. 

Population/Participants/Subj ects : 

Of the almost 125 teachers in our professional development program. 111 completed the 
knowledge mapping survey and 115 teachers completed and returned the evaluation of student 
work measure. The sixth grade teachers in this study worked in 25 public schools, in seven 
school districts in Southern California and Arizona. The teachers, on average, taught two classes 
of 25 students each. 

Intervention/Program/Practice : 

After an introductory professional development session, PowerSource treatment teachers and 
researchers met for three more professional development sessions of approximately 90 minutes 
each. During the first 45 minutes of each session, teachers and researchers discussed student 
work on the formative assessments associated with a particular foundational concept, possible 
misconceptions identified by those assessments and possible instructional interventions to correct 
those misconceptions. The last 45 minutes of each session focused on another single “big idea” 
(the meaning of multiplication or the meaning of equality) and its application, how that big idea 
would be developed from its nascent form into abstract concepts in algebra, and how the big idea 
could be appropriately taught and applied to sixth grade subject matter. To aid teachers with 
their instruction, teachers were given an instructional handbook on each of the big ideas during 
the second half of each 90 minute session. The professional development integrated this 
instructional handbook (pedagogical content) with the conceptual development of each of the big 
ideas (content knowledge). The teachers then returned to their classrooms to develop their actual 
instructional plan and to instruct their students on the applicable big idea for two class periods of 
approximately 40 minutes each. After the initial presentation of a big idea to their students, 
teachers were encouraged to continue to use each big idea in other instructional units they 
developed during the year. 

Research Design: 

Our study involved a randomized field trial where subject teachers were approximately equally 
divided between treatment and control groups. Treatment teachers received PowerSource 
professional development. Control teachers were further divided into three groups. The first 
control (in two districts) received the usual district professional development; the second control 
group (in one district) received instruction in determining the technical quality of district 
benchmark assessments, and the final control (in four districts) received instruction in student 
self-efficacy and motivation. 
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Data Collection and Analysis: 

All teachers in the treatment and control groups created a knowledge map and evaluated a 
sample of student work at the beginning of the study prior to any professional development. At 
the end of the school year, after the completion of treatment and control professional 
development activities, teachers again completed the same knowledge map and evaluation of 
student work activities. 

Each teacher knowledge map was compared to an expert knowledge map created by the 
researchers conducting professional development. This “expert map” was created by combining 
the individual maps of those researchers. Although the individual maps were identical on more 
than 98% of the relationships and concepts, the researchers met to resolve remaining differences 
in the map prior its use as the “expert” standard. 

The percentage of teacher agreement of the teacher map with the expert map was analyzed using 
a paired t-test to determine significance. More importantly, the mean agreement score of the 
treatment and control groups were analyzed using a t-test to determine the significance of the 
difference between these two groups. 

All teacher evaluations of student work (both pre- and post-PD) were scored by two raters based 
on the rubrics developed by Heritage and Vendlinski (2006). Differences in ratings given by 
these two raters were discussed by three other raters to arrive at a “true” score. 

The significance of change between the pre- and post-PD evaluations of student work were 
analyzed for each group using the Chi-square statistic. In addition, the significance of the 
differences between changes in the treatment and the control group were analyzed using this 
same statistic. 

Findings/Results: 

Initial results suggest that while there are differences between pre- and post-professional 
development knowledge maps and the ability of teachers to analyze the results of student work 
on formative assessments. While these trends are positive, the changes detected are small and are 
not significant at the a = .05 level. Initial analysis of the data from the evaluation of student work 
task are more positive and suggest that these improvements are correlated with participation in 
PowerSource professional development activities. 

Conclusions: 

Our results suggest that both the way teachers organize their content knowledge and the way 
they evaluate student work changes in important ways. Even with the positive trends, however, 
our findings suggest that nine hours of professional development may not be enough to produce 
the degree of change anticipated. As the literature suggests, producing change in teacher 
pedagogical practice requires sustained and intense efforts. These results suggest areas where 
important changes might be made such as a more intense focus on pedagogical methods, 
replacing in-situ curriculum with more conceptually appropriate instruction, and providing 
focused problems that teachers can use for sustained practice in their classrooms In addition, we 
intend to use more powerful Heirarchical Linear Models (HLM) to account for the degree of 
participation in professional development activities and to observe the degree of actual classroom 
implementation of PowerSource. 
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